Method and apparatus for filtering an image using a guidance image

ABSTRACT

A method and an apparatus for filtering an image using a guided image filter are described. A filter computes a filtering output using a guidance image. For calculating the filtering output confidence values associated to a filter input are taken into account. A confidence-weighted averaging is performed using the confidence values.

This application claims the benefit, under 35 U.S.C. §119 of European Patent Application No. 13306287.7, filed Sep. 20, 2013 and European Patent Application No. 13306804.9, filed Dec. 20, 2013.

FIELD OF THE INVENTION

The invention relates to a method and an apparatus for filtering an image, and more specifically to a method and an apparatus for filtering an image using a guided image filter.

BACKGROUND OF THE INVENTION

Recently a so-called Guided Image Filter has been developed as an edge-preserving smoothing filter. See He et al.: “Guided Image Filtering”, Proceedings of the 11th European conference on Computer vision (ECCV '10) (2010), pp. 1-14. Besides the input for the data to be filtered, the Guided Image Filter provides another input for a guidance image that drives the filtering process. It has been proven to be efficient in a wide range of applications, including alpha matting (see He et al.: “A Global Sampling Method for Alpha Matting”, Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR '11) (2011), pp. 2049-2056), and others. The Guided Image Filter has several key properties and advantages. A first advantage is the preservation of fine details in the filter input. A further advantage is that the filtered output has a higher quality than comparable filters. For example, it does not suffer from gradient reversal artifacts like the Bilateral Filter (see Tomasi et al.: “Bilateral Filtering for gray and color images”, Proceedings of the 1998 IEEE International Conference on Computer Vision (ICCV) (1998), pp. 839-846). Finally, the Guide Image Filter supports an extremely efficient implementation. Its complexity is O(n), where n denotes the number of pixels in the image. Notably, this means that the computational complexity is independent of the chosen size of the filter kernel. No other edge-preserving smoothing filter has this property.

A confidence-aware Bilateral Filter is disclosed in Jachalsky et al.: “Confidence evaluation for robust, fast-converging disparity map refinement”, 2010 IEEE International Conference on Multimedia and Expo (ICME) (2010), pp. 1399-1404. Besides potential gradient reversal artifacts, it suffers from computational complexity, especially for larger kernel sizes.

SUMMARY OF THE INVENTION

It is thus an object of the present invention to propose an improved solution for filtering an image using a guidance image.

According to the invention, this object is achieved by a method for filtering an image using a guided image filter, wherein a filtering output is computed using a guidance image. Confidence values associated to the filter input are taken into account for calculating the filtering output.

Accordingly, an apparatus configured to filter an image using a guided image filter comprises a filter configured to compute a filtering output using a guidance image, which is further configured to take confidence values associated to the filter input into account for calculating the filtering output.

Similarly, a computer readable storage medium has stored therein instructions enabling filtering of an image using a guided image filter, which when executed by a computer, cause the computer to compute a filtering output using a guidance image, wherein confidence values associated to the filter input are taken into account for calculating the filtering output.

In many applications, a confidence map accompanies the data to be filtered. A confidence value provides a measure of how reliable an element in the input data is deemed to be. The proposed solution extends the standard Guided Image Filter to take advantage of such confidence information. As a result the quality of output of the Guided Image Filter is improved by taking advantage of confidence data associated with the filter input. In particular, this is achieved without sacrificing the O(n) implementation complexity of the Guided Image Filter.

One element of the Guided Image Filter is a linear regression analysis. According to the invention, a weighted linear regression is employed where the confidence values serve as the weights, i.e. a confidence-aware Guided Image Filter is proposed.

As the Guided Image Filter is a smoothing filter that is able to preserve the fine details in the filter input, the Guided Image Filter is applicable in a broad range of applications. The confidence-aware Guided Image Filter is useful for any application where noisy data that is accompanied by confidence values is to be smoothed, e.g. alpha matting, disparity estimation, colorization, optical flow, and more. The proposed extended Guided Image Filter enables significant improvements in the quality of the final filter output with minimal computational overhead and while maintaining its O(n) complexity.

In case of alpha matting, for a training data set provided from www.alphamatting.com, the proposed confidence-aware Guided Image Filter can improve the quality (SAD) of the generated alpha mattes by about 28% on average when compared the standard version of the filter.

Compared to the confidence-aware Joint Bilateral Filter proposed by Jachalsky et al., the confidence-aware Guided Image Filter provides higher output quality.

Because of its low computational complexity and its low memory requirements, the confidence-aware Guided Image Filter is ideally suited for mobile devices.

For a better understanding, the invention shall now be explained in more detail in the following description with reference to the figures. It is understood that the invention is not limited to this exemplary embodiment and that specified features can also expediently be combined and/or modified without departing from the scope of the present invention as defined in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically shows a method according to the invention for filtering an image using a guided image filter, and

FIG. 2 illustrates an apparatus adapted to implement a solution according to the invention for filtering an image using a guided image filter.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In the following two exemplary implementations of the Guided Image Filter shall be described. In one formulation of the Guided Image Filter (the classical filter formulation), the filter output q is formulated as a weighted average of its neighboring pixels: q _(i)=Σ_(j) W _(ij)(I)p _(j).

Here, q_(i) is the filter output for pixel i, p_(j) denotes the filter input at pixel j, and W_(ij)(I) denotes the weight between pixels i and j, computed from the Guidance Image I. This formulates the output as a classical weighted average image filtering operation. Notably, the output of the Bilateral Filter of Tomasi et al. has the same formulation. Further, Jachalsky et al. disclosed a confidence aware extension of the Bilateral Filter, where the weights W_(ij) are modulated by the confidence values. Therefore, it might seem straight forward to adopt the same approach for the Guided Image Filter as well. Unfortunately, this classical filter formulation of the Guided Image Filter has a super-linear complexity, making such an approach less interesting. Therefore, the present invention focuses on the primary formulation of the Guided Image Filter which provides O(n) computational complexity. It is a goal of the present invention to maintain this linear complexity.

The following provides a brief overview of the primary formulation of the Guided Image Filter that provides an implementation complexity of O(n). More details can be found in He et al.: “Guided Image Filtering”, Proceedings of the 11th European conference on Computer vision (ECCV '10) (2010), pp. 1-14. Two cases are distinguished.

Case 1: The guidance image is a grey-scale image

The Guided Image Filter assumes that the filter output q is a linear transformation of the guidance image I in a window ω_(k) centered at pixel k: q _(i) =a _(k) I _(i) +b _(k) , ∀i∈ω _(k), where I_(i) denotes the pixel i in the Guidance Image, and a_(k) and b_(k) are some coefficients assumed to be constant in ω_(k). To determine the coefficients, the Guided Image Filter seeks a solution that minimizes the difference between the filter input p and its output q. The specific cost function is:

${{E\left( {a_{k},b_{k}} \right)} = {\sum\limits_{i \in \omega_{k}}^{\;}\;\left( {\left( {{a_{k}I_{i}} + b_{k} - p_{i}} \right)^{2} + {ɛ\;{a_{k}}^{2}}} \right)}},$ where ε is a regularization parameter preventing the coefficient a_(k) from becoming too large. The optimal coefficients can be computed via linear regression:

$a_{k} = \frac{{\overset{\_}{Ip}}_{k} - {{\overset{\_}{I}}_{k}{\overset{\_}{p}}_{k}}}{\sigma_{k}^{2} + ɛ}$ $b_{k} = {{\overset{\_}{p}}_{k} - {a_{k}{{\overset{\_}{I}}_{k}.}}}$

Here,

${\overset{\_}{Ip}}_{k} = {\frac{1}{\omega }{\sum\limits_{i \in \omega_{k}}^{\;}{I_{i}p_{i}}}}$ denotes the mean product of I_(i) and p_(i) in ω_(k), Ī_(k) and σ_(k) ² are the mean and variance of I in ω_(k), |ω| denotes the number of pixels in ω_(k), and p _(k) denotes the mean of p in ω_(k).

Since q_(i) is contained in more than one window ω_(k), the Guided Image Filter expresses the final filter output as the average output over all windows:

${q_{i} = {{\frac{1}{\omega }{\sum\limits_{k:{i \in \omega_{k}}}^{\;}\;\left( {{a_{k}I_{i}} + b_{k}} \right)}} = {{{\overset{\_}{a}}_{i}I_{i}} + {\overset{\_}{b}}_{i}}}},\mspace{14mu}{where}$ ${\overset{\_}{a}}_{i} = {\frac{1}{\omega }{\sum\limits_{k:{i \in \omega_{k}}}^{\;}a_{k}}}$ ${\overset{\_}{b}}_{i} = {\frac{1}{\omega }{\sum\limits_{k:{i \in \omega_{k}}}^{\;}{b_{k}.}}}$

Case 2: The guidance image is a 3-channel color image.

As for the grey-scale case, the Guided Image Filter assumes that the filter output q is a linear transformation of I in a window ω_(k)centered at pixel k: q _(i) =a _(k) ^(T) I _(i) +b _(k) , ∀i∈ω _(k), where I_(i) now is a 3×1 vector representing the color of pixel i in the Guidance Image, and a_(k) is a 3×1 coefficient vector for ω_(k). Again, a_(k) and b_(k) are assumed to be constant in ω_(k) and are computed by linear regression as a _(k)=(Σ_(k) +εU)⁻¹( Ip _(k) −Ī _(k) p _(k)) b _(k) = p _(k) −a _(k) ^(T) Ī _(k).

Σ_(k)denotes the 3×3 covariance matrix of I in ω_(k). Its elements can be computed as

${\Sigma_{k}^{({m,n})} = {{\frac{1}{\omega }{\sum\limits_{i \in \omega_{k}}^{\;}\;{I_{i}^{(m)} \cdot I_{i}^{(n)}}}} - {\frac{1}{\omega }{\left( {\sum\limits_{i \in \;\omega_{k}}^{\;}\; I_{i}^{(m)}} \right) \cdot \frac{1}{\omega }}\left( {\sum\limits_{i \in \;\omega_{k}}^{\;}\; I_{i}^{(n)}} \right)}}},$ where |ω| denotes the number of pixels in ω_(k), and I_(i) ^((x)) denotes the value of color channel x of I at pixel i.

As for the grey-scale case, since q_(i) is contained in more than one window ω_(k), the Guided Image Filter expresses the final filter output as the averaged output over all windows that contain pixel i:

${q_{i} = {{\frac{1}{\omega }{\sum\limits_{k:{i \in \omega_{k}}}^{\;}\;\left( {{a_{k}^{T}I_{i}} + b_{k}} \right)}} = {{{\overset{\_}{a}}_{i}^{T}I_{i}} + {\overset{\_}{b}}_{i}}}},\mspace{14mu}{where}$ ${\overset{\_}{a}}_{i} = {\frac{1}{\omega }{\sum\limits_{k:{i \in \omega_{k}}}^{\;}a_{k}}}$ ${\overset{\_}{b}}_{i} = {\frac{1}{\omega }{\sum\limits_{k:{i \in \omega_{k}}}^{\;}{b_{k}.}}}$

In the following a modified confidence-weighted linear regression will be described.

As can be seen above, the standard linear regressions in the Guided Image Filter do not take into account how reliable some input p_(i) is deemed to be. The regression line is computed under the assumption that all p_(i) are equally reliable. Any residuals are equally weighted. The proposed invention overcomes this restriction by computing a regression line that weights the residuals according to the confidence associated to p_(i). For many applications, such confidence data is generated in a preceding processing stage together with the data to be filtered, and is thus readily available. In the case of a grey-scale image the coefficients are computed by minimizing the following energy:

${{E\left( {a_{k},b_{k}} \right)} = {\sum\limits_{i \in \omega_{k}}^{\;}\left( {{c_{i}\left( {{a_{k}I_{i}} + b_{k} - p_{i}} \right)}^{2} + {ɛ\;{a_{k}}^{2}}} \right)}},$ where c_(i) denotes the confidence associated to the filter input p_(i). The computation of the weighted linear regression coefficients is known in the art. In general, any averaging operation is replaced by a weighted averaging operation. Since the present invention uses the confidence values as the weights, a weighted average value is computed as follows:

${{\overset{\sim}{\overset{\_}{X}}}_{k} = \frac{\Sigma_{i}X_{i}c_{i}}{\Sigma_{i}c_{i}}},$ where X refers to the quantity to be averaged.

The following describes the consequences for both the grey-scale case and the color case.

According to one aspect of the invention, the modified coefficients ã_(k) and {tilde over (b)}_(k) for the grey-scale case are computed as

$\overset{\sim}{a} = \frac{{\overset{\sim}{\overset{\_}{Ip}}}_{k} - {{\overset{\sim}{\overset{\_}{I}}}_{k}{\overset{\sim}{\overset{\_}{p}}}_{k}}}{{\overset{\sim}{\sigma}}_{k}^{2} + ɛ}$ ${{\overset{\sim}{b}}_{k} = {{\overset{\sim}{\overset{\_}{p}}}_{k} - {{\overset{\sim}{a}}_{k}{\overset{\sim}{\overset{\_}{I}}}_{k}}}},\mspace{14mu}{with}$ ${\overset{\sim}{\overset{\_}{Ip}}}_{k} = \frac{\Sigma_{i}I_{i}p_{i}c_{i}}{\Sigma_{i}c_{i}}$ ${\overset{\sim}{\overset{\_}{I}}}_{k} = \frac{\Sigma_{i}I_{i}c_{i}}{\Sigma_{i}c_{i}}$ ${\overset{\sim}{\overset{\_}{p}}}_{k} = \frac{\Sigma_{i}p_{i}c_{i}}{\Sigma_{i}c_{i}}$ ${\overset{\sim}{\sigma}}_{k}^{2} = {\frac{\Sigma_{i}{c_{i}\left( {I_{i} - \overset{\sim}{{\overset{\_}{I}}_{k}}} \right)}^{2}}{\Sigma_{i}c_{i}}.}$

Similarly, the modified coefficients ã_(k) and {tilde over (b)}_(k) for the color case are computed as

${\overset{\sim}{a}}_{k} = {\left( {{\overset{\sim}{\Sigma}}_{k} + {ɛ\; U}} \right)^{- 1}\left( {{\overset{\sim}{\overset{\_}{Ip}}}_{k} - {{\overset{\sim}{\overset{\_}{I}}}_{k}{\overset{\sim}{\overset{\_}{p}}}_{k}}} \right)}$ ${{\overset{\sim}{b}}_{k} = {{\overset{\sim}{\overset{\_}{p}}}_{k} - {{\overset{\sim}{a}}_{k}^{T}{\overset{\sim}{\overset{\_}{I}}}_{k}}}},\mspace{14mu}{with}$ ${\overset{\sim}{\overset{\_}{Ip}}}_{k} = \frac{\Sigma_{i}I_{i}p_{i}c_{i}}{\Sigma_{i}c_{i}}$ ${\overset{\sim}{\overset{\_}{I}}}_{k} = \frac{\Sigma_{i}I_{i}c_{i}}{\Sigma_{i}c_{i}}$ ${\overset{\sim}{\overset{\_}{p}}}_{k} = {\frac{\Sigma_{i}p_{i}c_{i}}{\Sigma_{i}c_{i}}.}$

Here {tilde over (Σ)}_(k) denotes the confidence-weighted covariance matrix of I in ω_(k). Its elements can be computed by

${\overset{\sim}{\Sigma}}_{k}^{({m,n})} = {\frac{\Sigma_{i}{I_{i}^{(m)} \cdot I_{i}^{(n)}}c_{i}}{\Sigma_{i}c_{i}} - {\frac{\Sigma_{i}I_{i}^{(m)}c_{i}}{\Sigma_{i}c_{i}} \cdot {\frac{\Sigma_{i}I_{i}^{(n)}c_{i}}{\Sigma_{i}c_{i}}.}}}$

Since the pixel q_(i) is contained in more than one window ω_(k), according to the standard Guided Image Filter, the coefficients of the individual windows are averaged to compute the final output q_(i). According to another aspect of the invention, the output q_(i) is computed by confidence-weighted averaging.

In one embodiment of the invention, the contribution of a window is weighted according to its window confidence, e.g. the sum (or equivalently the average) of the confidence values within the window. In this case, the coefficients are computed by

${\overset{\_}{\overset{\sim}{a}}}_{i} = \frac{\sum\limits_{k:{i \in \omega_{k}}}^{\;}{{\overset{\sim}{a}}_{k}c_{k}}}{\sum\limits_{k:{i \in \omega_{k}}}^{\;}c_{k}}$ ${{\overset{\_}{\overset{\sim}{b}}}_{i} = \frac{\sum\limits_{k:{i \in \omega_{k}}}^{\;}{{\overset{\sim}{b}}_{k}c_{k}}}{\sum\limits_{k:{i \in \omega_{k}}}^{\;}c_{k}}},\mspace{14mu}{with}$ $c_{k} = {\sum\limits_{i \in \omega_{k}}^{\;}\;{c_{i}.}}$

In another embodiment of the invention, the contribution of a window is weighted according to the confidence of its center pixel. In this case, the coefficients are computed by

${\overset{\_}{\overset{\sim}{a}}}_{i} = \frac{\sum\limits_{k:{i \in \omega_{k}}}^{\;}{{\overset{\sim}{a}}_{k}c_{k}}}{\sum\limits_{k:{i \in \omega_{k}}}^{\;}c_{k}}$ ${\overset{\_}{\overset{\sim}{b}}}_{i} = {\frac{\sum\limits_{k:{i \in \omega_{k}}}^{\;}{{\overset{\sim}{b}}_{k}c_{k}}}{\sum\limits_{k:{i \in \omega_{k}}}^{\;}c_{k}}.}$

Here, c_(k) denotes the confidence value associated to the center pixel in the window ω_(i).

The last five equations above hold for the grey-scale case and the color case. In the grey-scale case, a_(k), b_(k), ã _(i) and {tilde over (b)} _(i) are scalar values, whereas for the color case, they are 3×1 vectors.

For the grey-scale case, the final filter output is computed as q _(i) = ã _(i) I _(i) + {tilde over (b)} _(i). whereas for the color case, the final filter output is computed as q _(i) = ã _(i) ^(T) I _(i) + b _(i).

A method according to one embodiment of the invention for filtering an image using a guided image filter is schematically shown in FIG. 1. In a first step the image to be filtered is retrieved 10. Then a guidance image is obtained 11, which may be the image to be filtered or a different image. Then confidence values associated to the filter input are retrieved 12. The confidence values are taken into account for calculating 13 the filtering output. Finally, the calculated filtering output is made available 14.

FIG. 2 schematically illustrates one embodiment of an apparatus 20 adapted to implement a solution according to the invention for filtering an image using a guided image filter. The apparatus has an input 21 with one or more input units for retrieving 10 the image to be filtered, obtaining 11 a guidance image, and retrieving 12 confidence values associated to the filter input, e.g. from an attached storage or from a network. It is likewise possible that the confidence values are calculated by the apparatus based on other input data, or that more than one input is used for retrieving the various input data. Also, the data may be obtained from an internal storage of the apparatus 20. A filter 22 calculates 13 the filtering output, taking into account the confidence values. The resulting filtering output is made available 14 at an output 23. The filter 22 may likewise be implemented as software running on a processor. Also, the input 21 and the output 23 may be combined into a single bidirectional interface. 

The invention claimed is:
 1. A method for filtering an image using a guided image filter, comprising: retrieving the image to be filtered, a guidance image, and confidence values associated with the image to be filtered; computing a filtering output from filter input values derived from the image to be filtered and confidence values associated with the filter input values using a linear transformation of the guidance image in a window, wherein coefficients for the linear transformation are computed from confidence-weighted average values obtained for the window; and outputting the computed filtering output.
 2. The method according to claim 1, wherein computing the filtering output comprises computing a regression line that weights residuals according to the confidence values associated with the filter input values.
 3. The method according to claim 1, wherein the filtering output is calculated using a confidence-weighted averaging of coefficients of individual windows.
 4. The method according to claim 1, wherein a contribution of a window is weighted according to its window confidence.
 5. The method according to claim 4, wherein the window confidence is the sum or the mean value of the confidence values within the window.
 6. The method according to claim 1, wherein a contribution of a window is weighted according to a confidence of the center pixel of the window.
 7. An apparatus configured to filter an image using a guided image filter, the apparatus comprising; an input configured to retrieve the image to be filtered, a guidance image, and confidence values associated with the image to be filtered; a filter configured to compute a filtering output from filter input values derived from the image to be filtered and confidence values associated with the filter input values using a linear transformation of the guidance image in a window, wherein coefficients for the linear transformation are computed from confidence-weighted average values obtained for the window; and an output for the computed filtering output.
 8. The apparatus according to claim 7, wherein the filter is configured to compute the filtering output by computing a regression line that weights residuals according to the confidence values associated with the filter input values.
 9. The apparatus according to claim 7, wherein the filter is configured to compute the filtering output using a confidence-weighted averaging of coefficients of individual windows.
 10. The apparatus according to claim 7, wherein the filter is configured to weight a contribution of a window according to its window confidence.
 11. The apparatus according to claim 10, wherein the window confidence is the sum or the mean value of the confidence values within the window.
 12. The apparatus according to claim 7, wherein the filter is configured to weight a contribution of a window according to a confidence of a center pixel of the window.
 13. A non-transitory computer readable storage medium having stored therein instructions enabling filtering of an image using a guided image filter which, when executed by a computer, cause the computer to; retrieve the image to be filtered, a guidance image, and confidence values associated with the image to be filtered; compute a filtering output from filter input values derived from the image to be filtered and confidence values associated with the filter input values using a linear transformation of the guidance image in a window, wherein coefficients for the linear transformation are computed from confidence weighted average values obtained for the window; and output the computed filtering output.
 14. The non-transitory computer readable storage medium according to claim 13, wherein the instructions cause the computer to compute the filtering output by computing a regression line that weights residuals according to the confidence values associated with the filter input values.
 15. The non-transitory computer readable storage medium according to claim 13, wherein the instructions cause the computer to compute the filtering output using a confidence-weighted averaging of coefficients of individual windows.
 16. The non-transitory computer readable storage medium according to claim 13, wherein the instructions cause the computer to weight a contribution of a window according to its window confidence.
 17. The non-transitory computer readable storage medium according to claim 16, wherein the window confidence is the sum or the mean value of the confidence values within the window.
 18. The non-transitory computer readable storage medium according to claim 13, wherein the instructions cause the computer to weight a contribution of a window according to a confidence of a center pixel of the window. 